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(54) A distributed architecture and associated 
computation 

(57) tn a packet switching network, a distributed ar- 
chitecture provides efficient computation of routes in 
Quality of Service (QoS)-based routing scenarios. Us- 
ing a client-server model, only designated route sen/ers 
store and maintain a database containing the entire net- 
work topology so that each network node is not required 
to store and maintain the network topology. Client nodes 
maintain a cache containing pre-computed routes so 
that they can often make routing decisions autonomous- 
ly A client contacts a designated route server only when 
the client cannot obtain from its local cache a route to a 
given destination that meets the performance require- 
ments A client cache may contain pre-computed routes 
with designated QoS profiles to all destinations or to a 
subset of destinations. Route servers may also contain 
caches, which may contain pre-computed routes to all 
destinations in the network with all QoS profiles, or may 
contain only a subset of such routes. 

Each client node may also be provided with intelli- 
gence to learn, maintain and adapt local information 
based on the statistical usage of the network. Client 
caches may learn statically, i.e. the cache contains 
routes based on a QoS profile provided by the network 
service provider, or they may learn dynamically i.e., 
routes are modified based on ongoing network usage 
statistics The goal is to minimize the need to contact 
the route server as much as possible. Protocols are de- 
fined to maintain synchronization between the route 
server and its clients distributed across the network 
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protocols for efficient quality of service-based 

These protocols need to be observed to ensure that all 
nodes have the latest view of the network topology 
stored at the route server. 
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Description 

Rankaround nf the Invention 



r0001] The present Invention relates to the field of 
L..o,=wi.rhinn networks, e.g., Asynchronous Transfer 
Mode (ATM) networks. Specifically, the Invention relates 
to a distributed architecture and associated protocols for 
computing resource-sufficient routes capable of meet- 
inq a requested Quality of Service (QoS) level. The dis- 
closed methodologies are equally applicable to connec- 
tion-oriented networks observing QoS-based source 
routing and connectionless networks supporting multi- 
ple QoS levels 

[0002] Real-time multi-media applications demand 
explicit QoS guarantees from the underlying network 
To guarantee a given QoS level in the network, switching 
nodes or routers implement a QoS-based routing sys- 
tem that can observe explicit resource reservation. Such 
a system endeavors to find cost-effective resource-suf- 
ficient routes. 

[00031 A QoS-based routing system requires network 
nodes to gather and maintain network topology as well 
as resource availability information. Figure 1 shows 
such a network indicated generally by reference numer- 
al 10 Network nodes 12, 14, and 16. which are routers 
or switches, exchange Information about their internal 
resource availability including nodal-characteristics, 
link-attributes, and end-system reachability information, 
etc Databases 18, 20, and 22, connected to nodes 12, 
14 and 16, respectively, store such network topology 
and resource availability Information. A source node us- 
es the network topology and resource availability infor- 
mation to compute resource-sufficient routes to the des- 
tination node that can support the desired QoS. 
[0004] As the network grows, there are corresponding 
increases in the size of the topology database, the time 
required to determine a route, and the network control 
traffic overhead The network scalability problem asso- 
ciated with QoS routing has been addressed by the con- 
cept of hierarchical routing A hierarchical routing sys- 
tem divides the network into logical groups. Only nodes 
that belong to the same logteal group exchange detailed 
topology and link state information with each other A 
node outside the group only stores aggregate informa- 
tion about the group. 

[0005] To further scale down routing overhead re- 
quirements as the network grows, resource availability 
information is exchanged only periodically (e.g.. every 
several minutes) or at the occurrence of a significant 
change in resource availability (e.g., link/node failures, 
link utilization exceeding the 90% mark, etc.). Because 
resource availability information may be outdated, an in- 
termediate node may reject a call setup request In this 
case, the call retraces its path to the originating node, 
which then tries an alternate route. 
[0006] Even in hierarchical networks, route computa- 
tion requires considerable processing to find a cost-ef- 



fective and resource-sufficient path that can meet the 
requested QoS level. First, the route computation en- 
gine at a node needs to perform a topology search to 
find a low cost path while ensuring that each link (phys- 
5 ical or logical) in the path has the resources to accept 
the connection. In addition, topology database mainte- 
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requirements on switching nodes. Nevertheless, cur- 
rently Implemented QoS-based and best effort routing 
10 networks require each node to compute routes, store 
and maintain a topology database, and participate in the 
topology update process. The proposed architecture is 
tailored to distribute processing and memory require- 
ments to support both QoS-based and best effort rout- 
's ing. . , 

[0007] A client-server architecture, in which a server 
stores all topology infomiation and performs all route 
computation, would alleviate the processing and stor- 
age requirements of each node. Figure 2 shows a cen- 
20 tralized client-server network. Indicated generally by ref- 
erence numeral 22, for implementing route computa- 
tion Under this approach, single route server 24 stores 
all network topology and resource availability informa- 
tion In associated database 26. Clients 28, 30, and 32 
25 store no topology or routing information locally and 
therefore must communicate with server 24 every time 
they need to make routing decisions Although the ar- 
chitecture in Figure 2 removes the processing and stor- 
age requirements at each node as shown in Figure 1 , 
30 network congestion still occurs as each client node must 
contact the server to make routing decisions. Also, lim- 
ited processing power at the server may impede fast 
computation of routes and communication of those 
routes back to the client nodes. 
35 [0008] It is desirable, therefore, to provide a distribut- 
ed architecture that makes Intelligent use of processing 
elements distributed across the network to enhance per- 
formance of a routing system. Current QoS-based rout- 
ing standards (e g., Private Network to Network Inter- 
40 face (PNNI) defined by the ATM Forum) do not address 
route computation methods. Earlier proposed route 
caching methods (e.g., routing in an Internet Protocol 
(IP) network), which store some route information local- 
ly apply only to connectionless routing with no QoS 
45 guarantees. Cache tables in such schemes do not guar- 
antee that the selected hop satisfies the requested QoS 
requirement. It is even more desirable, therefore, to pro- 
vide a cllent-sen/er architecture for determining routes 
from a source node to a destinatbn address or node 
50 that can provide QoS guarantees along the route. 
[0009] The problem of route caching is more acute for 
networks that provide QoS guarantees because, in such 
an environment, the memory required to store cached 
routes may be prohibitively large For example, in a con- 
55 nection-oriented network observing QoS-based source 
routing, a cached route entry contains an entire descrip- 
tion of an end-to-end path to a given destination for a 
given QoS level On the other hand, a route that can 
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satisfy a given QoS level may not be able to meet an- 
other QoS requirement, and a source, therefore, rnay 
be required to store multiple QoS-based routes to the 
same destination. Similarly, a connectionless network 
may be designed to store information for the next hop, 
.,o,i^„= OnS Ifivels. Furthermore, with dy- 
namically changing resource availability inside the net- 
work, an end-to-end route description or the next hop 
route entry for a given QoS profile may not be valid after 
some time. Therefore, the cached routes need to be up- 
dated to ensure their validity 

roo10] Earlier proposed client-server approaches to 
assist QoS-based routing (e g.. Multiprotocol over ATM 
(MPOA), LAN Emulation (LANE), and Multicast Address 
Resolution Ser/er (MARS) protocols) only perform ad- 
dress resolution and do not address distributed rou e 
computation methods. It is also desirable, therefore, o 
provide a distributed route computation methodology to 
complement these client-server architectures Further 
there has been no effort to make use of distributed 
processing techniques to optimize perfomaance of a 
QoS-based routing system and enhance network scal- 
ability In such an environment. To further improve per- 
formance, it is also desirable to eliminate the need for 
every node in the network to participate in the topology 
exchange process and store network topology 

Summary the Invention 



room This invention satisfies those desires by pro- 
viding a distributed client-server architecture that allows 
fast access to routing information. Route servers, dis- 
tributed across the network, store and maintain a net- 
work topology database. Client nodes store pre-com- 
puted routes locally and access route sen/ers only when 
unable to obtain routing information locally. The present 
invention provides performance gains and enhances 
scalability in QoS-based networks observing source- 
based routing such as PNNI, MPOA, and LANE net- 
works. 

[00121 A method consistent with the present invention 
comprises the steps of searching a route cache at a 
source node for a route satisfying a QoS profile and ob- 
taining a route from a route sen/er if no route is found in 
the route cache. The method further compnses the step 
of updating the contents of the route cache based on 
network usage or changes in the state of the network 
Another method consistent with the present invention 
also comprises the step of populating the route cache 
with a plurality of pre-computed routes. Yet another 
method consistent with the present invention comprises 
the steps of searching a route cache at a source node 
for a route satisfying a QoS profile, searching a route 
cache at a route server for a route if no route is found in 
the source route cache, and computing a route at the 
route sen/er if no route is found in the server route 

[0013] Apparatus and networks are also provided for 



carrying out methods consistent with the present inven- 
tion 

[00141 The advantages accruing to the present inven- 
tion are numerous. For example, rather than storing all 
5 topology information on a centralized server, the inven- 
tive architecture provides a subset of routing information 

at client nodes, so mai a uhbih .iv~= — 

contact the route sen/er on every call request A client 
node needs to contact a route server only when infor- 
10 mation is not available locally Further, each individual 
client node has the intelligence to leam, maintain, and 
adapt local information based on the statistical usage of 
the network, minimizing as much as possible the need 
to contact the route server. Advantageously clients au- 
15 tonomously decide which subset of information to store 
locally Finally, the invention provides protocols for syn- 
chronizing client information and the topology database 
stored at the route sen/er. 

[0015] The above desires, other desires, features, 
20 and advantages of the present invention will be readily 
appreciated by one of ordinary skill in the art from the 
following detailed description of the preferred imple- 
mentations when taken in connection with the accom- 
panying drawings Both the foregoing general descrip- 
25 tion andthe following detailed description are exemplary 
and explanatory only and do not restrict the claimed in- 
vention The accompanying drawings, which are incor- 
porated In and constitute a part of this specification, il- 
lustrate several embodiments of the invention and to- 
30 gether with the description sen/e to explain the pnnci- 
ples of the invention. 



Rriof nflscriotion of the Drawings 

35 [0016] 

Figure 1 is a high level diagram of a fully distributed 
routing scheme known in the art; 

Figure 2 is a high level diagram of a prior art cen- 
tralized architecture for route computation; 

Figure 3 illustrates a networking scenario with dis- 
tributed route caching consistent with the present 
invention; 

Figure 4 illustrates an implementation of distributed 
route computation consistent with the present in- 
vention; and 

Figures 5A-B illustrate the operation of a protocol 
for distributed route computation consistent with the 
present invention. 

55 n^^tailed Descrip ti"" nf the Preferred Embodiments 

[0017] Figure 3 illustrates a network architecture con- 
sistent wrth the present invention in which network to- 
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pology databases are maintained and stored only at se- 
lected route servers distributed across the network. For 
this purpose, the whole network is divided into clus ers 
called routing tiers, three of which-tiers 40, 42, and 44- 
are shown for purposes of this discussion. Each routing 
A A ;«.^h.Hoc a rniJtG server-46, 48, ana 

tier 4u, cii " "^'"^ " " _i * 

50 respectively -that services a group of clients distnb- 
uted across an interconnection network By way of ex- 
ample, route server 46 sen^ices clients 58, 60, and 62, 
route sen/er 48 services clients 64, 66, and 68, and route 
sen/er 50 servicescllents70, 72, and74 Within the rout- 
ing tiers, only the route servers store and maintain the 
entire network topology, which is stored, for example in 
databases 52, 54, and 56 When the network upda es 
the topology databases, each server communicates 
with other servers to transfer Information pertaining to 
its clients Client nodes do not maintain their own copies 
of the topology database. As will be described in more 
detail below, client nodes maintain local caches, not par- 
ticularly shown in Figure 3. to store 
routes. Clients access sen/ers to obtain route infornr^a- 
tion only when pre-computed information is not availa- 
ble at the client's local cache. 

fOOIBl Route servers 46, 48, and 50 may be either 
network nodes, such as routers or switches, or separate 
processors which are not part of the network. The cli- 
ents, on the other hand, may be physical switches or 
routers or processor elements within a distributed 
swrtching system. In a distributed route caching archi- 
tecture consistent with the present invention, each clier^t 
communlcateswithltsservervia an Interconnection net- 
work The interconnection network can take many 
forms With continuing reference to Figure 3, one such 
Interconnection network Is shared bus 76, connectirig 
clients 58, 60, and 62 to route server 46. Shared bus 76 
may connect nodes within the same physical location or 
may operate on the backplane of a distributed switching 
system. A second type of interconnection network is a 
Local Area Network (LAN) such as an Ethernet network, 
an FDDI network, or a token ring network. For example, 
in Figure 3, LAN 80 interconnects client nodes 70, 72, 
and 74 with route server 50. A third interconnection net- 
work Wide Area Network (WAN) 78, Interconnects Gh- 
ent nodes 64, 66, and 68 with route sen/er 48 A distrib- 
uted network caching methodology consistent with the 
present invention is not limited to a specific networking 
scenario or type of interconnection network, and the in- 
terconnection networks illustrated in Figure 3 are exem- 
plary only. 

100191 When a network node receives a call setup re- 
quest, attempts to find the best route to the specified 
destination address or destination node. The setected 
route must meet the bandwidth requirements of the call 
as well as the required QoS level. Routes are usually 
optimized based on user-specified criteria, e g Admin- 
istrative Weight (AW), end-to-end delay, and end-to-end 
delay variation, either alone or in combination. 
[0020] A distributed network routing methodology 



consistent with the present invention selects optimal 
routes without duplk;at,ng the entire topology database 
at each individual network node Instead, each node 
contains a cache for storing a set of pre-cornputed 
5 routes These routes are classified with respect to their 
destination addresses or nodes, bandwidth require- 

ment Virtual Hrivaie iMeiwuiMuci.i...v-. -.-^ 

end-to-end delay requirement, delay variance con- 
straints, and cell loss ratio requirement), and service 
10 category (e.g., real-time or best effort, continuous b, 
rate or variable bit rate). These route attributes used for 
route classification are collectively referred to herein as 
the QoS prolile When a source node receives a call 
destined for an address or node for which a suitable 
rs route meeting the desired QoS profile is available in the 
cache, the node forwards the call along the rou e re- 
irieved from the cache. If the client node cannot ind a 
suitable route in the cache, the client node solicits the 
route sen/er to obtain a route to the desired destination^ 
20 ro021l A route caching methodology consistent with 
the present invention reduces latency in route selection 
by providing from the cache a suitable route for an in- 
comingcall setup request. The number of pre-computed 
routes that can be stored in the cache is limited by the 
25 amount of memory that can be dedicated to route cach- 
ing at a single node. Furthermore, the pre-compu ed 
cached routes must be based on the latest view of the 
network topology stored at the route ^e-ver. These con- 
straints must be considered in determining the contents 
30 of each client node's cache. 

r00221 The following equation is useful in analyzing 
the dynamic of a distributed architecture consistent with 
the present invention: 



35 



Mean Route Access Time = A'h + 8(1 -h). 



(1) 



r0023] In equation (1 ), ^ represents the average time 
required to find a suitable route in the client node s local 
40 cache, B represents the average time to fetch a route 
from the route se^er when the client node cannot find 
a route locally, and h represents the cache hit rate, de- 
lined as the fraction of requests for which the client node 
can find a cached route. When the client node must con- 
45 tact the route server to obtain a route, the route server 
may have to perform route computation on demand, so 
that generally Bis significantly greater than A 
r00241 Therefore, one objective of a distributed route 
caching scheme consistent with the present invention is 
50 to maximize h at each node The hit rate is maximized 
by caching a network corridor map at the client node, 
where the network corridor map is a table containing 
routes from the client (source) node to every possible 
destination address for a given QoS profile. Caching 
55 network corridor maps for multiple QoS profiles increas- 
es the hit rate further A distributed route caching 
scheme consistent with the present invention allows 
network service providers to define QoS profiles for 
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which routes should be cached based on their past ex- 
perience with operating the network 
[00251 While distributed route caching methodologies 
consistent with the present invention strive to maximize 
the hit rate, h, the memory constraints of storing pre- 

. , „..t,„inHK/iHnalrlientslsalsocx)nsid- 

compuiea luuico Qi un- ... . 

ered Depending on the size of the network and the 
amount of cache memory available at a client node, it 
may not be feasible to cache a QoS-based route to eve- 
ry destination of local interest. When limited cache 
memory is available, clients cache only routes to a sub- 
let of destinations of local interest with the QoS profiles 
of local interest For purposes of further discussm a 
client node will be referred to herein as fully cached rf 
the client caches QoS-based routes to all destinations 
of local interest with all the QoS profiles of local interest 
A client node will be referred to herein as partially 
cached if the client caches routes only to a subset of 
network destinations of local interest. Distributed route 
caching methodologies consistent with the Present in- 
vention allow network service providers to define the 
destinations and QoS profiles of local interest. 
r00261 In a distributed route caching methodology 
consistent with the present invention, there are at leas 
two alternatives for establishing the locality of interest 
at each client node: (1) static learning, in wh.ch each 
client uses the locality of interest defined by the network 
sen/ice provider and does not modify the local interest 
based on network usage; and (2) dynamic learning, in 
which each client uses the localrty of interest defined by 
the network service provider to populate the cache ini- 
tially but modifies the destinations and QoS profiles of 
local interest based on network usage. With dynamic 
learning, the localities of interest of the client nodes may 
vary temporally (e.g. , by time of the day or day of weelj 
as well as spatially (i.e., various nodes may commonly 
request calls to different sets of nodes or addresses in 
the network). Route caching consistent with the presen 
ir^vention explorts each node's patterrn of P-^t ««;yP°^J 
and spatial behavior to determine a locality of nterest 
torthe node at any point in time By utilizing past behav- 
ior each node predicts which routes it is likely to need 
in the near future and stores these routes In its cache, 
[00271 The performance of distributed route caching 
schemes consistent with the present invention can be 
further maximized by reducing the latency, B, to fetch a 
route from the route server when the client node cannot 
find a route locally For this purpose, in addition to main- 
taining local caches at the client nodes, a distributed ar- 
chitecture consistent with the present invention may 
maintain a cache at the senrer node as well. The cache 
at the server complements the cache information pro- 
vided to the individual clients. For example, the individ- 
ual clients may observe partial caching only for routes 
of local interest, while the server may use a full cache 
which contains routes to all destinations with all QoS 
profiles supported by the network sen/ice provider^ 
[0028] A full cache at the server consistent with the 



present invention caches network corridor maps with all 
QoS profiles supported by the network service provider. 
On the other hand, a fully cached client stores only 
routes to destinations of local interest for the QoS pro- 
5 files of local interest. Therefore, distributed route cach- 
ing methodologies consistent with the present invention 
may operate in at leasi onsj ui luu, — - 
caching at both the sender and clients, and (2) full cach- 
ing at the seiver and partial caching at the clients, (3) 
w full caching at the clients and partial caching at the serv- 
er, and (4) partial caching at both the seiver and clients 
[0029] Thus, an architecture and methodotogy con- 
sistent with the present invention determine the locality 
of interest of each client node, manage the limited cache 
15 space to suit the local interest, and synchronize the 
cache with the latest view of the topology database so 
that pre-computed routes remain valid. 
r0030] By way of example only. Figure 4 details a spe- 
cific software implementation of a distributed architec- 
20 ture consistent with the present inventton. However, one 
skilled in the art will appreciate that there exist several 
implementations consistent with the present Invention, 
and the invention is not limited to the specific implemen- 
tation shown. Shared bus 100 interconnects route serv- 
25 er 102 with client switching nodes 104, 106, and 108, 
all of which operate within one routing tier. Alternatively, 
the shared bus may be another type of interconnection 
network e g , a LAN or WAN as discussed in connection 
with Figure 3. The client switching nodes may also be 
30 routers, and route sen/er 102 may be an additional 
switching node or may be a separate processor which 
does not route or switch traffic. Each switching node 
104 106 and 108 contains a route cache for storing pre- 
computed routes 110, 112, and 114, respectively, a 
35 cache manager 1 1 6, 1 1 8, and 1 20, respectK^ely, a each- 
ing agent 122, 124, and 126, respectively, and a routing 
aqent 128 130, and 132, respectively. Ftoute sewer 102 
includes network topology database 134, synchronizer 
136 route dispatcher 138, background route computa- 
40 lion engine 140, on-demand route computation engine 
142, topology database manager 144. topology ex- 
changer 1 46, and route cache 1 48. 
[0031] cache managers 116, 118, and 120 may be 
software entities resident on each node. Each cache 
45 manager controls the usage of local memory available 
for route caching by using one or more of the following 
functions: (1) learning, i e., determining the population 
of the cache at start-up and which new routes should be 
added on an ongoing basis; (2) route replacement i.e., 
50 determining which route ent^r should be removed from 
the cache when the cache is full and a new route entry 
must be added: and (3) route purging, i.e., determining 
when and why entries should be removed, separate 
from the route replacement function A client node can 
65 perform these operations at a low priority e g., the client 
may accumulate multiple caching requests and bundle 
them together in a single message to the route server. 
[0032] Each cache manager determines the popula- 
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10 



tion of the cache using prior knowledge of the usage pat- 
tern of calls originating from the client node. The network 
service provider defines an initial QoS profile for each 
client node, characterized by the destination address, 
service category, bandwidth requirement, and required 
r.^o i„„»i cnmp subset thereof. The cache manager 
may use the initial QoS profile to populate the cache with 
pre-computed routes. 

[0033] The cache manager may increase its knowl- 
edge of the node's locality of interest on an ongoing ba- 
sis by tracking the call setup requests the node receives. 
When the client obtains a route from the server because 
Its route cache contains no routes meeting the request- 
ed QoS profile and optimization criteria, the cache man- 
ager places the newly obtained route in its cache. Sim- 
ilarly, the cache manager updates the route cache as a 
result of crankback, a PNNI feature known in the art that 
enables a blocked call setup request to return to the 
source or an earlier intermediate node in its path, which 
attempts to reroute the call. When a node learns that a 
call setup has failed for a route previously selected by 
the node, the node Informs its cache manager to inval- 
idate the route. If the node then obtains a new route from 
the server, the cache manager replaces the new route 
in the cache in place of the old, invalid route 
[00341 When the cache manager learns about a new 
QoS profile and attempts to add it to the client's route 
cache, the cache manager may remove an entry to free 
memory for the newly learned route. There are several 
approaches a cache manager consistent with the 
present Invention may take, alone or in combination, in 
deciding which entry to replace. According to one ap- 
proach the cache manager maintains records of the last 
time an entry in the cache has been hit, i.e., the lasttime 
the client node selected the entry as an optimal route. 
With this method, the cache manager replaces the entry 
which has not been hit for the longest period of time. 
According to another approach, the cache manager 
maintains a count of the number of times an entry has 
been hit i e , the number of times the cache manager 
has selected the entry as an optimal route. With this 
method, the cache manager replaces the entry that has 
received the smallest number of hits. According to yet 
a third approach, the cache manager maintains both a 
count of the number of hits and the time elapsed since 
the last hit. The cache manager then weights the count 
of hits by the time elapsed since each hit so that the 
weight of a particular hit on an entry decreases with time. 
In one implementation of this weighted approach, an en- 
try in the route cache has a holding priority during the 
kth time Intewal represented by; 



(2) 



where , is the number of hits the given route 
received during the {k-1)s\ interval, and the weight w is 



a number such that 0 <= w <= 1 If the cache manager 
must make a replacement decision during the time pe- 
riod p the cache manager replaces the entry with the 
lowest H value. If two or more entries share the lowest 
5 value, the cache manager selects an entry to be re- 
placed using some criteria, e.g., by randomly selecting 
one of the candidate entries. 

[0035] Even when the cache manager does not need 
to remove entries from the cache because the cache is 
10 underutilized, it may be desirable to purge any cached 
routes that have remained in the cache for a significant 
period of time. Similarly, when a route suffers a crank- 
back, it may be purged. 

[0036] Consistent with the present invention, the cu- 
ts ent node may purge routes autonomously or with assist- 
ance from the route seiver, as will be described below 
In one approach to autonomous purging, the client node 
places an upper limit on the lifetime of a cached route, 
regardless of how often the node obtains the route di- 
20 rectly from the route cache. Upon expiration of the route 
lifetime, the cache manager purges the route from the 
cache A purged route may be automatically fetched 
again from the route server if certain criteria are met, e. 
g il the purged route was popular in the recent past, as 
25 measured by comparing its hit rate to a predefined hit 
rate threshold. 

[0037] With continuing reference to Figure 4, caching 
agents 122, 124, and 126 may be software entities res- 
ident on each client node. Each caching agent stores, 
30 removes, and retrieves cached routes on a demand ba- 
sis Each caching agent is optimized for efficient retriev- 
al of the cached routes based on the destination ad- 
dress QoS profile, and optimization criteria. For exam- 
ple, each caching agent may use a data structure that 
3S can efficiently search the cache to find a route that sat- 
isfies the QoS requirements and optimization criteria to 
a given destination. In addition to matching QoS profiles 
with routes in the local route cache, the caching agent 
may implement some additional functions. For example, 
40 if for a given destination address and QoS profile there 
are multiple routes available in the cache, the caching 
agent may achieve load balancing by using a technique 
such as weighted round robin to select one route over 
others based on certain criteria. Another function of the 
45 caching agent is to determine whether to use a cached 
route whose QoS profile surpasses that of the call re- 
quest in order to avoid contacting the route server to 
obtain a route. For example, the caching agent may de- 
cide to use a cached route capable of supporting 2 Mbps 
50 bandwidth even though the connection requires only 1 5 
Mbps bandwidth. 

[0038] Routing agents 1 28, 1 30, and 1 32 are software 
modules on each node that interpret a call request to 
extract the destination address or node and QoS profile. 
55 Based on this information, a routing agent prepares a 
route request message for the caching agent. If a route 
cannot be obtained from the cache the routing agent 
solicits the route server for a suitable route When the 
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routing agent obtains a route from the route seo/er, the 
routing agent inforrr^s the cache manager so that he 
cache manager can learn the new route and add it to 
the client node's route cache. 

[00391 Synchronizer 136 in route sen/er 102 main- 

*■ . I — ttA^r^ru- tnnolnnv database 1 34 

tains syncnroriizciuui i ^--i . 1 — 

and route caches 1 1 0 J 1 2. and 1 1 4 at the client nodes^ 
For this purpose, the synchronizer tracks usage of 
routes by the distributed clients. If the client nodes im- 
plement autonomous purging, then, consistent with the 
present Invention, route server 102 tracks route usage 
by remembering, forapenod of time equal tothe lifetime 
of the route before purging occurs, that a particular client 
node has stored a route in its route cache. If the route 
undergoes a change before the route lifetime expires (e^ 
a if a node becomes inactive and must be removed 
from all routes, or if cached routes become invalid as a 
result of a resource availability update or crankback and 
no longer satisfy the required QoS profile), the route 
sen/er informs the client of the change, instructs the ch- 
ent to purge the route, and discontinues updates for that 
route to the client. If the client determines that it should 
continue to store the route in its route cache, it fe ches 
the route again from the route server Alternatively in- 
stead of instructing the client to purge the route, the 
route server can provide the client with the newly com- 
nuted route, which would remain in the clients route 
cache for the lifetime of the route. In either case, once 
the route lifetime has expired, the route server assumes 
that the client has already purged the route, and discon- 
tinues updates pertaining to that route. If the client has 
already purged or replaced the route before the route 
lifetime has expired, the client will ignore the route serv- 
er's updates. This overall approach, in which the rou e 
server remembers which routes are stored in each cli- 
ent's route cache, ensures that locally stored routes are 
always synchronized with the route server's view of the 
network, as stored in its network topology database. 
100401 In a second method for tracking route usage 
consistent with the present invention, the client does not 
perform autonomous purging of routes from its route 
cache, instead, the client depends on the route server 
to track the routes stored in the local route cache and to 
refresh them periodically The route sen/er maintains i^s 
knowledge of routes in the local route caches either by 
periodically probing the clients or by being in orrned by 
the clients when an existing route is replaced. Under the 
first scenario, there may be a period of time between 
probes during which the local caches may not be syn- 
chronized with the route server Under the second sce- 
nario however, the local caches are synchronized with 
the route sen/er's topology database The second ap- 
proach, however, requires more processing by both the 
client node and the route server 
roOAIl When a client node cannot find a route in its 
route cache that satisfies a given call setup request, the 
routing agent of the client contacts the route sen/er to 
obtain a route Route sen/er 102 receives all routing re- 



quests through route dispatcher 1 38, which is respon^ 
sible for instructing route computation engines 140 and 
1 42 to compute routes in the background or on demand 
respectively Background route computation engine 1 40 

5 typically computes network corndor maps or routes cur- 
rently cached by the client nodes to populate individual 

clients on a periodic basis, un-u^ma, ^ " 

tion engine 142 typically computes a route when a client 
cannot find a route locally and contacts the route server 
10 to obtain a route. 

r0O42] When route dispatcher 1 38 receives a request 
lor a route, it forwards this request to on-demand route 
computation agent 142. Alternatively, if route sen/er 102 
also maintains its own route cache of pre-computed 
,5 routes, as illustrated by route cache 148 in Figure 4, 
route dispatcher 138 may search the servers route 
cache before soliciting an on-demand route computa- 
tion Maintaining independent cache 148 at the server 
is advantageous when the client nodes have limited 
zo memory and/or the sen/er has a large memory space 
tor route caching. With its own route cache, the serve 
may store network corridor maps containing optimal 
routes to all destinations with several QoS profiles to 
avoid on-demand path computation as much as possi- 
25 ble sen/er route cache 148 is maintained in the same 
Lnner as other caches In the system. Whether route 
dispatcher 138 obtains a route from a computation en- 
gine or the server's route cache, it returns to the client 
a cost-effective and resource-sufficient route to the giv- 

30 en destination. ir, 
10043] Topology database manager 144, shown in 
Figure 4, maintains network topology database 134 
stored at route server 102 and receives updates frorri 
topology exchanger 146. Route sen/ers across a net- 
as work exchange information through topology exchanger 
entities, which typically participate In the topology up- 
date process on behalf of the clients in its routing tier. 
This architecture reduces the processing needed at c i- 
ent nodes by removing them from the topology update 

40 process. , „ ... 

[00441 Figures 5A-B illustrate the operation of a dis- 
puted route computation protocol consistent wrth the 
present invention undertwo scenarios. In Figure 5A, the 
desired route is obtained from the local cache^arid in 
45 Figure 5B, a desired route is not available in the local 
cache in Figure 5A, the client node receives a route re- 
quest at its routing agent The routing agent then pre- 
pares a route request for the caching agent, including 
the destination address, QoS profile, and optirriization 
so criteria The caching agent retrieves a suitabte route 
from the local route cache and infomns the cache man- 
aqer of the route hit so that the cache manager can 
maintain statistics of route usage. The caching agent 
delivers the retrieved route to the routing agent, which 
ss passes the route to the switch or router 

[00451 Figure 58 illustrates retrieval of a route from 
the route server The routing agent receives a route re- 
quest and passes the destination address, QoS profile. 
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and optimization criteria to tine caching agernt^ I the 
caching agent cannot find a suitable route in the loca 
cache the caching agent Informs the routing agent of 
the failure. The routing agent then contacts the route 
server through the interconnection network (e.g^, 
.... tho declination ad- 
shared bus, LAIN, or vvniM;, h>°oc,..m 

dress, QoS profile, and optimization criteria to the serv- 
er The route server obtains a suitable route, which it 
transmits back to the routing agent In the chent^The 
routing agent informs the cache manager and caching 
agent of the new route so that the newly obtained route 
can be added to the cache. Finally, the routing agent 
delivers the route to the switch or router. 
[00461 It will be appreciated by those skilled in this art 
hat various modifications and variations can be made 

to the distributed architecture and protocols consistent 
with the present Inventbn described herein w-thout de- 
parting from the spirit and scope of the invention. Other 
embodiments of the Invention will be apparent to those 
skilled in this art from consideration of the specincation 20 

and practice of the invention disclosed herein. For ex- 
ample Figure 4 illustrates only an exemplary implemen- 
tation of software modules, and many other Implemen- 
tations consistent with the present invention are possi- 
ble In the methodology discussed in connection with 
Figure 4, clients obsen/e partial caching and dynarnical- 
ly learn about the locality of interest based °n network 
usage However, several combinations of caching (full 
orpartialattheclientsand/orsen/er)andcachelearning 

(Static and dynamic) are consistent with the present in- 
vention. For example, when destination addresses and 
QoS profiles are static, the cache manager does not use 
leaming algorithms. Similarly, if a client node stores 
routes ?o every possible destination of local interest for 
the QoS profiles of Interest, the cache manager need 
no replace routes to free memory for storing newly 
learned routes. Instead, the client simply replaces the 
old entry with the newly learned ent^. Additionally, han- 
dling of crankbacks may be simplified where the client 
node labels the route whk=h has suffered the failure and 40 
either uses an alternate route from Its own cache or re- 
quests the se^er for an alternative route. If a new route 
is obtained from the sen/er. the client node simply up- 
dates the labeled route with the newly learned entiy. The 
server, on the other hand, may trigger recomputation of 
the network corridor map, possibly in the background, 
in the event of a crankback failure and/or on a periodic 
basis. The route server may bundle updates for a whole 
network corridor map in one message to sinnjily the 
synchronization process Therefore, it is intended tf^at 
the specification and examples be considered exempla- 
ry only with the true scope and spirit of the invention 
being indicated by the following claims. 



selecting a route between a source node and a des- 
tination node, wherein the route satisfies a QoS pro- 
file wherein the source node stores in a route cache 
a plurality of pre-computed routes originating at the 
source node, and wherein the source node is con- 
nected to a route sen/er, the method comprising the 
steps of 



ss 



Claims 

1 . A method, for use in a packet switching network, for 



searching the route cache tor a route satisfying 
the QoS profile; and 

if no satisfying route is found in the route cache, 
obtaining a route from the sen/er satisfying the 
QoS profile. 

The method of claim 1 further comprising the step 

°' updating the contents of the route cache 
based on network usage. 

The method of claim 2 wherein the step of updating 
Includes: 

adding the route computed at the server to the 
route cache 

The method of claim 3 wherein the step of updating 
further Includes 

if the route cache is full before the computed 
route Is added, removing one of the pre-computed 
routes from the route cache. 

The method of claim 3 wherein the step of updating 
further includes; 

for each pre-computed route In the route cache, 
measuring time since the pre-computed route 
was selected as the route between the source 
node and the destination node; 

comparing the measured times; 

if the route cache is full before the computed 
route is added, choosing one of the pre-com- 
puted routes to be removed from the route 
cache based on the comparison of measured 
times; and 

removing the chosen one of the pre-computed 
routes from the route cache. 

The method of claim 3 wherein the step of updating 
further includes; 

for each pre-computed route In the route cache, 
maintaining a count of the number of times a 
pre-computed route is selected as the route be- 
tween the source node and the destination 
node; 
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comparing the counts; 

if the route cache is full before the computed 
route is added, choosing one of the pre-com- 
puted routes to be removed from the route 
_ - - 1 A ^r. tho rnmnarifion of counts; and 

CclUt lt=! uaoc\-i ^--i " " ^ 1 

removing the chosen one of the pre-computed 
routes from the route cache. 

The method of claim 3 wherein the step of updating 
further includes: 

for each pre-computed route in the route cache, 
measunng time since the pre-computed route 
was selected as the route between the source 
node and the destination node; 

for each pre-computed route in the route cache, 
maintaining a count of the number of times a 
pre-computed route is selected as the route be- 
tween the source node and the destination 
node; 

weighting the counts by the measured times; 25 
comparing the weighted counts; 



10. The method of claim 3 wherein the step of updating 
further includes 

for each pre-computed route in the route cache, 
measuring time since the pre-computed route 
was added to the route cache; 

choosing one of the pre-computed routes to be 
purged from the route cache based on the 
measured time; and 

purging the chosen one of the pre-computed 
routes from the route cache. 
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if the route cache is full before the computed 
route is added, choosing one of the pre-com- 
puted routes to be removed from the route 
cache based on the comparison of weighted 
counts; and 

removing the chosen one of the pre-computed 3S 
routes from the route cache. 

8 The method of claim 7 wherein the step of weighting 
the counts by the measured times includes the sub- 
step of 



calculating a holding priority during the kth time 
interval according to the equation = w*k.i + 
(1-w) Uk.i where Uk-i is the count during the 
(k-l)st interval, and w is a number such that 0 

<iz w 1 ; 

and wherein the step of choosing one of the 
pre-computed routes to be removed includes 
the substep of 

during the time period p, choosing the one of 
the pre-computed routes with the lowest Hp. 



The method of claim 3 wherein the step of updating 
further includes: 

tor each pre-computed route in the route cache, 
measuring time since the pre-computed route 
was added to the route cache: 

comparing the measured times to a predeter- 
mined route lifetime; and 

purging from the route cache the pre-computed 
routes whose measured time exceeds the route 
lifetime. 

12. The method of claim 11 wherein the step of updating 
further includes: 

for each pre-computed route in the route cache, 
maintaining a count of the number of times a 
pre-computed route is selected as the route be- 
tween the source node and the destination 
node; 

updating the purged route based on a current 
state of the network; and 

adding the updated route to the route cache if 
the count before purging exceeds a threshold. 

13 The method of claim 1 wherein the route server 
maintains a database of the network topology, the 
method further comprising the step of: 

synchronizing the route cache with the net- 
work topology database. 

50 14. The method of claim 13 wherein the step of syn- 
chronizing includes the substep of: 

the route server probing the source node to 
determine the contents of the route cache. 



40 



45 



■ tho ctpn 55 15 The method of claim 1 further comprising the step 

9. The method of claim 1 further comprising the step lb. me m 

of: ^ , uDdatinq the routes stored in the route cache 

purging a pre-conpuled route fror. the route ^^^^pda.ng ^ ^^^^^^ ^^^^^^^ 

cache 
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16. The method ot claim 1 further comprising the step 

updating the routes stored in the route cache 
in response to a failure in the network. 

rv,^^h.^H riaim 1 further comprising the step 

If. I I 1^ I I iv.*" ^' 

updating the routes stored in the route cache 
in response to a crankback in the network. 

18. The method of claim 1 further comprising the step 
of- 

updating the routes stored in the route cache 
in response to a change in availability of a resource 
in the network. 



15 



1 9. A method, for use in a packet switching network, for 
selecting a route between a source node and a des- 
tination node, wherein the route satisfies a QoS pro- 
file, and wherein the source node is connected to a 
route server, the method comprising the steps of: 

populating a route cache at the source node 
with a plurality of pre-computed routes originat- 
ing at the source node, 

searching the route cache for a route satisfying 
the QoS profile; and 

if no satisfying route is found in the route cache, 
obtaining a route from the server satisfying the 
QoS profile. 

20 In a packet switching network including a plurality 
of route servers and a plurality of client network 
nodes, wherein each client node is connected to a 
route server, wherein each client node stores in a 
route cache a plurality of pre-computed routes orig- 
inating at the client node, wherein the route server 
stores in a server route cache a second plurality of 
pre-computed routes, a method for selecting a route 
between a first node and a second node that satis- 
fies a QoS profile, the method comprising the steps 
of: 

searching the client route cache for a route sat- 
isfying the QoS profile; 

If no satisfying route is found in the client route 
cache, searching the server route cache for a 
route satisfying the QoS profile; and 

if no satisfying route is found in the server route 
cache, computing a route at the server that sat- 
isfies the QoS profile. 

21 In a packet switching network including a plurality 
of route servers and a plurality of client network 



20 



25 



35 



40 



45 



nodes, wherein each client node is connected to a 
route server, and wherein the route server main- 
tains a database containing network topology and 
resource availability information to be used for com- 
puting routes that satisfy a QoS level, a method for 
updating the database comprising the steps of: 

each route server maintaining network topolo- 
gy and resource availability information; and 

each route server exchanging with the remain- 
der of the plurality of route servers network to- 
pology and resource availability information 
about the clients connected to it. 

22. A packet switching network comprising: 

a route server; 

a source node connected to the route server; 
a destination node; 

a route cache connected to the source node, 
the route cache containing a plurality of pre- 
computed routes originating at the source 
node; and 

means for selecting a route between the source 
node and the destination node that satisfies a 
QoS profile, the means comprising: 

means for searching the route cache for a route 
satisfying to the QoS profile; and 

means for obtaining a route from the sen/er that 
satisfies the QoS profile if no satisfying route is 
found in the route cache. 

23 The network of claim 22 further composing means 
' for updating the route cache based on network us- 
age. 

24. The network of claim 23 wherein the means for up- 
dating includes: 

means for adding the route computed at the 
server to the route cache. 
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25. The network of claim 24 wherein the means for up- 
dating further includes: 

means for removing one of the pre-computed 
routes from the route cache if the route cache is full 
before the computed route is added. 

26. The network of claim 24 wherein the means for up- 
dating further includes: 

means for measuring time since each pre-com- 
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puted route was selected as the route between 
the source node and the destination node; 

means for connpanng the nneasured tinnes; 

means for choosing one of the pre-computed 
routes to be removed from the route cache 
based on the comparison of measured times if 
the route cache is full before the computed 
route is added; and 

means for removing the chosen one of the pre- 
computed routes from the route cache. 

27. The network of claim 24 wherein the means for up- 
dating further includes: 

means for maintaining a count of the number of 
times each pre-computed route is selected as 
the route between the source node and the des- 
tination node; 

means for comparing the counts; 

means for choosing one of the pre-computed 
routes to be removed from the route cache 
based on the comparison of counts if the route 
cache is full before the computed route is add- 
ed; and 

means for removing the chosen one of the pre- 
computed routes from the route cache. 

28. The network of claim 24 wherein the means for up- 
dating further Includes: 

means for measuring time since each pre-com- 
puted route was selected as the route between 
the source node and the destination node; 

means for maintaining a count of the number of 
times each pre-computed route is selected as 
the route between the source node and the des- 
tination node; 

means for weighting the counts by the meas- 
ured times; 

means for comparing the weighted counts; 

means for choosing one of the pre-computed 
routes to be removed from the route cache 
based on the comparison of weighted counts if 
the route cache is full before the computed 
route is added; and 

means for removing the chosen one of the pre- 
computed routes from the route cache. 



29. The network of claim 28 wherein the means for 
weighting the counts by the measured times in- 
cludes 

5 means for calculating a holding priority during 

the kth time interval according to the equation 
Hl = w*Hl.i +{1-w) Uu.i where Uu.i is the count 
during the (k-1)st interval, and w is a number 
such that 0 w <= 1 ; 

10 

and wherein the means for choosing one of the 
pre-computed routes to be removed includes 

during the time period p, means for choosing 
is the one of the pre-computed routes with the 

lowest Hp, 

30. The network of claim 23 further comphsing: 

means for purging a pre-computed route from 
20 the route cache. 

31 . The network of claim 24 wherein the means for up- 
dating further includes: 

2S for each pre-computed route in the route cache, 

means for measuring time since the pre-com- 
puted route was added to the route cache; 

means for choosing one of the pre-computed 
30 routes to be purged from the route cache based 

on the measured time; and 

means for purging the chosen one of the pre- 
computed routes from the route cache. 

35 

32. The network of claim 24 wherein the means for up- 
dating further includes: 

for each pre-computed route in the route cache, 
40 means for measuring time since the pre-com- 

puted route was added to the route cache; 

means for comparing the measured times to a 
predetermined route lifetime, and 

45 

means for purging from the route cache the pre- 
computed routes whose measured time ex- 
ceeds the route lifetime. 

50 33. The network of claim 32 wherein the means for up- 
dating further includes: 

for each pre-computed route in the route cache, 
means for maintaining a count of the number of 
55 times a pre-computed route is selected as the 

route between the source node and the desti- 
nation node; 
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means for updating the purged route based on 
a current state of the network; and 

means for adding the updated route to the route 
cache if the count before purging exceeds a 
threshold. 

34. The network of claim 23 wherein the route server 
maintains a database of the network topology, the 
network further comprising: 

means for synchronizing the route cache with 
the network topology database. 

35. The network of claim 34 wherein the means for syn- 
chronizing includes: 

means for probing the source node to deter- 
mine the contents of the route cache. 

36. The network of claim 22 further comprising: 

means for updating the routes stored in the 
route cache in response to a change in the network. 

37. The network of claim 22 further comprising: 

means for updating the routes stored in the 
route cache in response to a failure in the network. 

38. The network of claim 22 further comprising: 

means for updating the routes stored in the 
route cache in response to a crankback in the net- 
work. 

39. The network of claim 22 further comprising: 

means for updating the routes stored in the 
route cache in response to a change in availability 
of a resource in the network. 

40. A packet switching network comprising: 

a route server; 

a source node connected to the route server; 
a destination node; 

a route cache connected to the source node; 

means for populating the route cache with a 
plurality of pre-computed routes originating at 
the source node; and 

means for selecting a route between the source 
node and the destination node that satisfies a 
QoS profile, the means comprising: 

means for searching the route cache for a 
route satisfying to the QoS profile; and 

means for obtaining a route from the server 



that satisfies the QoS profile if no satisfying 
route is found in the route cache. 

41. A packet switching network comprising: 

5 

a plurality of route servers; 

a plurality of client network nodes, each client 
node connected to a route server; 

10 

a route cache connected to each client node, 
the route cache containing a plurality of pre- 
computed routes originating at the client node; 

15 a route cache connected to each route server 

containing a second plurality of pre-computed 
routes; 

means for selecting a route between a first 
20 node and a second node that satisfies a QoS 

profile, the means comprising: 

means for searching the client route cache for 
a route satisfying the QoS profile; 

25 

means for searching the server route cache for 
a route satisfying the QoS profile if no satisfying 
route is found in the client route cache; and 

30 means for computing a route at the server that 

satisfies the QoS profile if no satisfying route is 
found in the server route cache. 

42. A packet switching network comprising: 

35 

a plurality of route servers; 

a plurality of client network nodes, each client 
node connected to a route server; and 

40 

a network topology and resource availability 
database, used to compute routes that satisfy 
a QoS level, connected to each route sen/er, 

45 wherein each route server comprises: 

means for maintaining network topology 
and resource availability information; and 

50 means for exchanging topology and re- 

source availability information pertaining to 
the clients connected to the route server 
with the remainder of the plurality of route 
servers. 

55 

43. A packet switching network comprising: 

a route server: 
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a plurality of source nodes connected to the 
route server; 

a plurality of destination nodes; 

a source route cache connected to each source 
node, the source route cache containing a pre- 
computed route from the source node to each 
destination node of local interest; and 

means for selecting a route between the source 
node and the destination node that satisfies a 
QoS profile, the means comprising: 

means for searching the source route 
cache for a route satisfying the QoS profile; 
and 



means for obtaining a route from the server 
satisfying the QoS profile if no satisfying 
route is found in the route cache. 



44. A packet switching network comprising: 
a route server; 

a plurality of source nodes connected to the 
route server; 



24 

a plurality of source nodes connected to the 
route server; 

a plurality of destination nodes; 

a source route cache connected to each source 

node, ine source ruuits ua»-i ^w. .i^ » - r ■ - 

computed route from the source node to a des- 
tination node in the network satisfying each of 
a plurality of QoS profiles of local interest; and 

means for selecting a route between the source 
node and the destination node that satisfies 
one of the plurality of QoS profiles, the means 
comprising: 

means for searching the source route 
cache for a route satisfying the QoS profile; 
and 

means for obtaining a route from the server 
satisfying the QoS profile if no satisfying 
route is found in the route cache. 



30 



a plurality of destination nodes; 

a source route cache connected to each source 
node, the source route cache containing a pre- 
computed route from the source node to a des- 
tination node in the network; 

a sen/er route cache connected to the route 
server, the server route cache containing a 
route from each source node connected to the 
route sen/er to each destination node in the net- 40 
work; and 

means for selecting a route between the source 
node and the destination node that satisfies a 
QoS profile, the means comprising: ''^ 

means for searching the source route 
cache for a route satisfying the QoS profile; 
and 



means for obtaining a route from the server 
satisfying the QoS profile if no satisfying 
route is found in the route cache, 

45. A packet switching network comprising 



25 46. A packet switching network comprising: 



a route server; 

a plurality of source nodes connected to the 
route server; 

a plurality of destination nodes; 

a source route cache connected to each source 
node, the source route cache containing a pre- 
computed route from the source node to a des- 
tination node; 

a server route cache connected to the route 
sen/er, the server route cache containing a 
route from a source node connected to the 
route server to a destination node in the net- 
work satisfying each of a plurality of QoS pro- 
files provisioned in the network; and 

means for selecting a route between the source 
node and the destination node that satisfies a 
QoS profile, the means comprising: 

means for searching the source route 
cache for a route satisfying the QoS profile; 
and 

means for obtaining a route from the server 
satisfying the QoS profile if no satisfying 
route is found in the route cache. 



so 
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a route server: 
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